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ABSTRACT 

This paper describes how visual media technologies 
have been used to support research into the way technical diagrams 
are represented in people's minds. Two groups of subjects 
participated in this research, individuals with either a high or a 
low level of expertise in meteorology, specifically professional 
meteorologists and non-meteorologists. Two general types of visual 
media technologies — computers and video“were used to investigate the 
mental representation of diagrams. Two main types of data were 
collected by the computer and video methodologies: (1) process data, 
the actions subjects perform as they work through a task; and (2) 
product data, the outcomes (drawings) of copying, recalling, or 
completing a weather map. Examples of the way process and product 
data were handled during analysis are presented. The strength of 
video technology is that it provides for the capture of highly 
detailed visual data. However, it also presents problems with the 
time-consuming nature of transcribing the data into a form suitable 
for statistical analysis. In contrast, computer technology can 
directly monitor some aspects of a subject's performance and save the 
generated data in files that are suitable for statistical analysis, 
although it is not currently capable of directly providing (for 
reasonable cost and effort) the type of facilities for dealing with 
visual data that are available with video. Recent merging of the two 
technologies with the advent of digital video is opening up new 
possibilities for research. (Contains 15 references.) (AEF) 
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• Points ol view or opinions stated in this This paper describes a selection of 

Soer?pos,I!o“oVp«^^ that visual media technologies have 

been used to support various aspects of 
research into the way technical diagrams 
are represented in people's minds. Its 
purpose is to demonstrate how such 
technologies can address some of the 
difficulties inherent in conducting research 
in this area. The research programme for 
^ which the techniques described here were 

Q devised has the broad goal of 

W understanding how learners deal with 

instructional diagrams. 



accessible to us for the purposes of study 
(unlike the outcomes of a diagram-based 
instructional task) and so its 
characterization presents a considerable 
research challenge. As a result, the 
methodologies involved in such diagram 
research of necessity rely on indirect 
approaches that aim to provide fine- 
grained data from which inferences can be 
made about mental representation. This 
requires that these approaches are not only 
robust and powerful, but also that they are 
appropriate to use with pictorial material. 




While illustrations in general can be 
included in instructional materials for a 
range of different purposes (Levie & 
Lentz, 1982), a central role of diagrams of 
the type used in technical domains is to 
explain some aspect of the subject matter 
(Levin, Anglin & Carney, 1987; Lowe, 
1993). To date, much investigation of 
expository illustrations such as diagrams 
has focussed upon their instructional 
outcomes, rather than upon the processes 
by which these outcomes are generated. 
However, the rise of information 
processing approaches to the study of 
thinking and learning in recent years has 
highlighted the need to understand mentf»l 
processes in trying to improve 
instructional outcomes. With this has come 
an increasing interest in the mental 
processes involved in the comprehension 
of pictorial materials such as diagrams 
(e.g. Winn, 1991). Fundamental to these 
mental processes is the way in which the 
information upon which they operate is 
represented in our minds. Consequently, 
before we can fully understand how 
diagrams are processed, we need to know 
about the mental reprccentation that people 
develop for the graphic elements which are 
the fabric of those diagrams. 
Unfortunately, a person's mental 
representation of a diagram is not directly 
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A fundamental consideration in 
designing research methodologies for this 
area arises from the nature of the materials 
to be investigated. Diagrams and similar 
explanatory visual materials typically are 
used because they are considered better 
suited to the presentation of the subject 
matter than verbal formats (written or 
spoken text). It now seems clear that 
aspects of the information structure of 
diagrams can provide them with cognitive 
processing advantages over text (Glenberg 
& Langston, 1992; Larkin & Simon, 
1987; Winn & Li, 1989). However, the 
great majority of research into the way 
people mentally represent information 
involves the use of verbal materials and 
verbally-oriented methodologies. There 
has been considerable development in the 
collection of detailed verbal data in recent 
years and approaches to analysis such as 
those discussed by Ericsson and Simon 
(1984) are well established. However, the 
direct adoption of such methodologies for 
the investigation of the way people deal 
with visual materials would require visual- 
to-verbal translations, so compromising 
their intrinsic ^ lal character. In other 
words, collecting verbal data about the 
processing of visual material (for example, 
think-aloud protocols generated by a 
subject performing a diagram processing 
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task) would be ol' dubious value on its 
own (although it may be a valuable source 
of supplementary data). What is required 
instead are approaches that properly 
acknowledge the visual nature of these 
materials and deal with them without the 
need for any translation. So for an 
investigation of the way people deal with 
diagrams, it is not sufficient to confine the 
visual aspect of the procedure to the 
stimulus materials. It is important that the 
outputs that subjects produce should also 
be visual as should any intervening 
manipulations of the materials they 
perform. The need' to use a consistently 
visual approach across the various aspects 
of investigation guided the development of 
the mcth<xlologies described in this paper. 
This development was informed by the 
approaches used by researchers in the 
investigation of cognitive aspects of 
various visually-oriented domains such as 
chess (Chase & Simon, 1973), 
architectural drawings (Akin, 1986), and 
electronic circuit diagrams (Egan & 
Schwartz, 1979). 

Before dealing with specific aspects 
ol' the methodologies that have been 
developed for the current research 
program, some general considerations 
regarding this type of research will be 
discussed. For a methodology to be useful 
lor fine-grained investigation of the way 
people interact with diagrams, it needs to 
be able to: 

1. Display the diagrammatic 
material effectively to the research 
.subjects, 

2. Provide appropriate ways for the 
subject to interact with the diagram, 

3. Capture detailed information 
about subject-diagram interactions, 

4. Characterize these subject- 
diagram interactions with appropriate 
measures, 

5. Permit generated data to be 
analyzed in a meaningful way. 



6. Deal with static and dynamic 
components of the investigation. 

As will be illustrated below, a 
suitable combination of the capacities of 
video and computer technology can 
address these types of needs in a variety of 
ways. In addition, the capacity of these 
technologies for dealing with dynamic data 
gives them a distinct advantage over more 
conventional (static) approaches to the 
reporting of results from this type of 
investigation. 

Diagrams and Subjects 

All the methodologies described here 
were applied to the study of how weather 
map diagrams are mentally represented. 
However, it should be noted that weather 
maps were used purely as illustrative 
examples of the type of diagram 
commonly used for instructional purposes 
(there was no interest in the investigation 
of weather maps per se). A discussion of 
the advantages of using weather maps for 
this purpose appears elsewhere (Lowe, 
1990). In principle, the methodologies are 
applicable to a variety of diagram types 
from a wide range of technical disciplines. 
The two groups of subjects who 
participated in this research were 
individuals with either a high or a low 
level of expertise in the domain of 
meteorology, specifically professional 
meteorologists and non-meteorologists. 
Because it involved comparison of two 
groups with greatly differing levels of 
domain expertise, this research approach 
can loosely be described as fitting within 
the "expert-novice" paradigm (Chi, 
Glaser, & Farr, 1988). However, because 
the meteorologists were competent 
practitioners rather than acknowledged 
expert performers in their domain, the 
expert-novice characterization should be 
treated with some caution. 

Data Collection 

In the following sections, the use of 
two general types visual media 
technologies (computers and video) for the 
investigation of the mental representation 
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of diagrams will be discussed. Although 
these approaches were specifically 
developed for use with very abstract 
diagrams of a technical nature, there is no 
reason in principle why they could not be 
adapted for various other types of 
graphical ly-simple pictorial materials that 
are used as instructional illustrations. 

Computer Technology 

The computer-based methodology 
described here is an adaption and 
extension of an earlier manual technique 
that has been reported in detail elsewhere 
(Lowe, 1989). In general terms, the 
methodology was used to investigate a 
subject’s mental representation for weather 
map diagrams by monitoring the way the 
subject explored an unfamiliar example of 
such a diagram which was initially hidden 
from view. This monitoring reli^ on the 
diagram having been segmented so that the 
subject’s exploration across individual 
regions could be followed. A 
superimposed 6x5 grid divided the 
weather map into 30 square pieces (the 
diagram consisted of a map outline and its 
meteorological markings). When a subject 
began the experimental task, the display 
was completely covered with squares that 
were blank except for the map outline 
(i.e., no meteorological markings). 
During the task, the subject gradually 
revealed parts of the display by removing 
the covering squares from the display one 
at a time. This allowed determination of 
the order in which the selected regions of 
the display were inspected. However a 
condition of the task was that the subject 
was permitted to expose a total of only one 
third of the whole display (10 of the grid 
squares) and told that from this limited 
amount of information, she/he would later 
be required to complete the display. 

In the computer implementation of 
this technique, the experimental task 
involved the subject interacting with a 
screen depicting a blank weather map of 
Australia and divided into 30 squares by a 
grid as described above (the map and grid 
were drawn in black on a white 
background). The subject was prompted 



through the required interactions by 
appropriate instructions provided in a side 
bar on the screen. To reveal what was 
under a particular square, the subject 
moved the computer’s mouse pointer onto 
that square and clicked once, upon which 
the square turned black to signal that it had 
been selected. The subject was then able to 
either confirm the selection with a second 
click on the blackened square, or select a 
different square by clicking on it instead. 
The second (confirmatory) click on a 
square caused the hidden meteorological 
markings on that part of the weather map 
to be revealed. For any square exposed in 
this way the background was changed 
from white to grey so that the subject 
could easily tell that it had already been 
revealed (this was necessary since not all 
of square segments comprising the map 
actually contained weather map markings). 
As this revelation process continued, the 
instruction side bar gave the subject 
updated information about how many of 
the quota of 10 squares remained available 
for selection. Once all 10 of the allowed 
squares had been revealed, the subject’s 
map showing the exposed meteorological 
markings was printed off. The subject then 
attempted to complete the whole weather 
map by drawing in any further 
meteorological markings that were thought 
to exist in the remaining 20 grid squares. 

Because the interactions just 
described are somewhat involved, each 
subject was trained in the necessary 
operational processes before moving on to 
the experimental task. The sequence of 
presentation was as follows. After an 
introduction to basic processes such as 
moving and clicking the mouse, the 
subject was given preliminary training in 
the technique of revealing information by 
clicking on squares. This was done using 
a simple four-square grid (Figure 1). 

Next, these skills were applied to a 
graphic display of a familiar scene (a 
simple house and its surroundings) 
following the same procedures that would 
be required for performing the 
experimental task with the weather map. 
The subject began with a bare outline of 
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the house divided by a grid of 30 squares 
(Figure 2) then by using the procedure 
described above for the experimental task, 
clicked on 10 squares in sequence to reveal 
one third of the house's details (windows, 
doors, etc.). 

For this practice task, the subject's 
response was not printed off for 
aimplction. After this practice, the subject 
carried out the experimental (weather map) 
task with weather map. During the 
experimental task, the computer program 
recorded the identity of the squares chosen 
and the sequence in which they were 
selected. Finally, a screen dump of the 
subject's partly completed map (showing 
the 10 revealed squares) was printed off 
for the subject to complete (Figure 3). 

This computer version of the original 
manual method was developed in 
HyperCard so that it was able to run on 
even low-end black and white Macintosh 
computers, thus making it suitable for use 
in a wider range of situations. Although 
some aspects of the methodology (such as 
the training segment) were altered to make 
the technique less dependent on the 
experimenter, the course of the 
experimental task remained fundamentally 
unaltered. The computer-based version has 
a number of benefits over the original 
pr(x:edurc such as the automatic collection 
of data regarding which squares were 
chosen by a subject and their order of 
.selection. It also has the potential to be 
used for displays other than weather maps 
as will now be explained. In the weather 
map version described above, the map 
outline constitutes one level of information 
(provided for the subject) and the set of 
markings that depict meteorological 
information constitutes another (to be 
provided by the subject). However, since 
the aimputer program is essentially a shell 
into which a particular display is fitted, the 
method could easily be adapted for 
investigating the way subjects search other 
diagrams that can be characterized in terms 
of two levels of information. For example, 
in the area of biology, subjects could be 
presented with an outline of an unfamiliar 
animal's body and the way they searched 



for hidden details of the animal's digestive 
system studied. Similarly, in the area of 
electronics, subjects could search for 
information about the components present 
in a hidden section of a circuit diagram. 

Video Technology 

The methodology described here w^s 
developed to obtain finer details about the 
mental representation of weather map 
diagrams than could be obtained from the 
technique described above. In addition, its 
basic approach to data collection was 
fundamentally different, thus providing the 
opportunity for triangulation. A video 
recorder was used to capture the sequence 
of actions performed by subjects in 
carrying out tasks involving a previously 
unseen Australian weather map. A major 
part of the process involved subjects 
drawing a copy of the map's 
meteorological markings onto a blank 
map. However, the weather map they 
were to copy was not continuously visible. 
Instead, subjects could only view the 
stimulus map while they were pressing a 
button with their drawing hand. This 
meant that they could either take a look at 
the stimulus map or draw markings onto 
the blank map but not continuously view 
and draw at the same time. Hence, 
subjects were forced to alternate viewing 
and drawing in a succession of discrete 
actions. An indicator light was switched 
on when the viewing button was pressed 
and remained illuminated for the duration 
of each viewing period (i.e., as long as the 
viewing button was depressed). 

The subject carried out the 
experimental tasks at a specially 
constructed table that had a glass window 
inserted into its top with a mirror set 
beneath the window at an angle of 45 
degrees. A video recorder aimed at the 
mirror allowed each subject's drawing 
behavior to be recorded from below 
without obstruction (the response sheet 
containing the blank map was made from 
tracing paper so that the subject's drawing 
was visible from below). The viewing 
indicator light was positioned so that it 
could be included in the area being 
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recorded by the video and the camera was 
fitted with a counter that marked each 
frame with elapsed time. Thus in addition 
to recording the subject's drawing of each 
meteorological marlang, the length of each 
viewing period was automatically recorded 
on videotape. As well as drawing a copy 
of the weather map, subjects later drew 
what they could recall of the map's 
markings. Video records of this process 
were also made to enable comparison of 
the intake of diagram information (during 
copying) with its ouQjut (during recall). . 

Data Analysis 

Two main types of data are collected 
by the techniques that have been 
described. The first are process data which 
come from the actions subjects perform 
during an investigation as they work 
through a task. Of particular interest here 
are the temporal order in which the 
meteorological markings are processed, 
the location of those markings, and the 
durations of various stages that comprise 
the processing. The second type are 
product data such as the outcomes 
(drawings) of copying, recalling, or 
completing a weather map. 

Analyzing each type of data presents 
its own challenges, some of which 
concern practicalities such as how to carry 
out particular measurements while others 
concern more fundamental theoretical 
issues involving the choice, treatment, and 
interpretation of those measurements. 
Some examples of the way process and 
product data were handled during analysis 
will now be presented. 

Process Data 

A number of assumptions were made 
here. First, it was assumed that the 
temporal order in which the meteorological 
markings were processed was in some 
respects influenced by the nature of the 
mental representation of the weather map 
information. This assumption parallels that 
made by Taylor and Tversky (1992) in 
their investigation of the mental 
organizations of environments. As these 



Figure 1. 

Grid used for uaining subjects in procedure for 
selecting squares (black indicates square 'A' has 
been clicked once; a further click would reveal 
what was under this square). 
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Figure 2. 

Practice task graphic showing outline within 30 
square grid. Note instructions to subject in side 
col umn . 




Figure 3. 

Screen dump showing markings on a subject's 10 
selected squares. Subject would next attempt to 
complete the map' markings. 
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authors have observed, one of the research 
difficulties in using drawings as data is 
how to score them. Taylor and Tversky 
used the order in which elements of a map 
arc drawn as an index of organization on 
the assumption that the clustering of items 
in a recall sequence occurs because those 
items are closely related in some way. In 
the video-based weather map 
investigations referred to in the present 
paper, this type of clustering assumption 
was made about the sequences produced 
during the copying of items as well as 
during their recdl. In the computer-based 
investigation mentioned earlier, 
assumptions about clustering were made 
from the order in which the 10 squares 
were selected for revelation. Depending on 
the particular task a subject was 
performing, the structure of these 
sequences may provide an indication of 
matters such as the effects of established 
habits of perceiving and drawing, the 
extent of visual impact (in perceptual 
terms) of different typiss of markings, the 
tendency of subjects to group the markings 
because of similarities in their visuospatial 
characteristics, the ease with which 
markings of different types were 
processed or the relative meteorological 
importance attached to different markings. 

The second assumption was that the 
relative durations of particular aspects of 
processing (such as the lengths of viewing 
periods) were of significance. In the 
current context, this assumption concerns 
only the video-based investigation 
described above. On most occasions 
during the copying task, subjects viewed 
the stimulus diagram using a series of 
relatively short glimpses. After each 
glimpse, a subject typically drew one or 
more marks. However, at various stages 
during the process of copying the stimulus 
diagram, the subject would spend a much 
longer time gazing at the diagram before 
changing to drawing. These longer 
viewing periods were interpreted as 
indicating the start of a new phase of 
processing in which the subject's attention 
shifted to a different group (or chunk) of 
graphic elements within the stimulus 
diagram. This interpretation permitted each 



subject's processing sequence to be used 
to identify subsets within the complete set 
of graphic elements comprising the 
diagram. By examining the constitution of 
these subsets, inferences could be made 
about inter-element relations that might be 
involved in the subject's mental 
representation of the diagram as a whole. 
For example, it appeared that the non- 
meteorologists structured the mental 
representation of information in the 
diagram on the basis of relations involving 
the visuospatial characteristics of the 
markings themselves whereas the 
meteorologists' structuring was based 
upon relations that reflected the 
meteorological significance of these 
markings. 

A major disadvantage of the process 
aspect of the video-based investigation 
was the time consuming and labour- 
intensive nature of the data transcription 
required. The raw data consisted of a 
videotape record of each subject's viewing 
and drawing behavior in which each 
videotape frame was marked with the time 
elapsed since the start of the task. In 
addition, the videotape of the copying task 
also contained records of the periods 
during which the viewing indicator light 
was switched on and switched off. 
Transcription of the videotape involved 
locating precise frames at which (a) the 
viewing indicator light was switched on or 
was switched off and (b) the subject's 
pencil just began to draw a mark or just 
finished drawing that mark. The elapsed 
time for each of these frames was 
transferred manually to build up a table 
that set out each subject's viewing and 
drawing behavior. In addition, each of the 
marks drawn on the map was labelled with 
its position in the overall drawing 
sequence. In contrast to the video-based 
investigation, the computer-based 
investigation described earlier 
automati^ly collected process data about 
the identity of the 10 squares that a subject 
revealed and the order in which they were 
selected. These data required no pre- 
treatment after collection and were ready 
for immediate transfer to a statistical 
analysis program. However, the 
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computer-based approach did not offer the 
same opportunities for collecting the subtle 
types of data that can be captured by the 
use of video. Clearly an integration of the 
capacity of video technology to collect 
detailed visual data and the automatic data 
collection capacity of computer technology 
would be the ideal combination. 

Product Data 

Map markings drawn by subjects 
and the larger graphic structures form^ by 
the conjunction of drawn and existing 
markings can be characterized by various 
measures. These measures allow a range 
of comparisons to be made including those 
between (a) the markings on the stimulus 
materials and those produced in the 
subjects' responses, (b) different classes 
of subject (such as experts and novices) 
and (c) products generated at different 
stages of an investigation (such as copied 
and recalled markings). Measures of 
position, size, shape, and orientation are 
useful to characterize the properties of 
individual markings both in terms of their 
basis visuospatial properties and in terms 
of the meteorological significance of those 
markings. For example, the sizes and 
relative positions of markings in of a set of 
concentric closed isobars that together 
constitute a cyclone not only show the 
physical extent of that feature, they also 
indicate its likely severity. As well as 
making measurements of each marking on 
its own, it is important to characterize 
larger composite structures made up of 
two or more markings which constitute 
meteorologically significant assemblies. 

A fundamental matter to be resolved 
in developing an analysis procedure is the 
definition of what constitutes a discrete 
entity for the purpose of analysis. In cases 
v.'here a marking consisted of a single 
graphic stroke or a set of physically 
connected strokes, this generally did not 
constitute a problem since it was usually 
quite clear that a single entity was 
intended. However, for other markings 
such as a cold front symbol, physically 
connected but geometrically distinct 
markings were considered as separate 
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entities because pilot studies showed that 
these different components were drawn by 
subjects as a number of separate 
subgroups (in the case of a cold front, a 
line plus a group of triangular barbs). 

Once individual graphic entities have 
been defined, the next problem is in 
making measurements of the position, 
size, shape, and orientation of each entity. 
Recent developments in image analysis 
computer software have greatly simplified 
this measurement task. These programs 
process an image that has been converted 
to digital form and then use various 
algoriSims to characterize properties of the 
image. In the case of the weather map 
diagrams used for the research referred to 
here, the image was captured by means of 
a flatbed scanner to form a disc file (in a 
format such as PICT). Some preliminary 
treatment using image enhancement 
software was usually necessary to clean up 
the scanned image and to prepare it for 
analysis. For example, regions bounded 
by particular sets of lines might be filled 
with "paint" to isolate them so that the 
image analysis program can later identify 
them as distinct features to be measured 
(each such region is typically described as 
a "blob"). Once the prepared image is 
loaded into the analysis program, the user 
can then specify the desired measurements 
and select the blobs of interest. Simple 
measures of distance (such as length or 
perimeter) and area are made by counting 
pixels which are then converted into more 
familiar units. The position of a blob is 
based upon its center of gravity while a 
variety of measures such as symmetry, 
roundness, and the ratio of the axes of the 
best fitting ellipse help to characterize its 
shape. For orientation, a measure of the 
inclination of the longest axis of the best 
fitting ellipse can be used. Data from these 
measurements are readily transferred to 
statistics packages for further processing. 

Reporting Results 

A combination of the capacities of 
computer and video technologies has 
enabled the findings obtained from the 
types of investigations described above to 
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be displayed in a manner that is more 
appropriate than conventional methods. 
This is particularly well demonstrated in 
the case of results obtained from analysis 
of the process data where the nature of the 
subjects' performance over time was of 
special interest. For example, a computer 
animation program was used to produce 
dynamic summaries of essential 
differences in how those in the 
meteorologist and non-meteorologist 
subject groups built up the meteorological 
markings as they copied a weather map. 
These summaries collapsed the individual 
marking sequences produced by subjects 
within these two groups into an overall 
sequence for each group as a whole. In 
addition, the animation greatly compressed 
the time scale so that the broad temporal 
patterns present in the sequence became 
much more apparent. Once prepared, the 
animated sequence for each of the subject 
groups was transferred to videotape for 
easy presentation and distribution. This 
animated form of the results captures 
process aspects of the subjects' 
performance far more faithfully than is 
possible in a static figure of the type 
typically used to report the findings of an 
investigation. Further, complex results of 
the type generated in this investigation 
appear to be considerably more accessible 
if presented in an animated format than if 
presented in a more conventional manner. 

Conclusion 

This paper has dealt with several 
illustrative examples of how video and 
computer technology can be used to 
support basic research into the way 
diagrams are represented in people's 
minds The approaches that have been 
developed can address a variety of 
challenges that face the researcher in this 
field, ranging from those of presenting the 
initial stimulus material to subjects through 
to reporting the research findings in an 
appropriate manner. In common with other 
research methodologies, in order to m^e 
use of these approaches effective, 
sufficient attention must be given to 
theoretically significant aspects of data 
collection, analysis, and interpretation. 



The strength of video technology is that it 
provides for the capture of highly detailed 
visual data. However, it also presents 
problems with the time-consuming nature 
of some aspects of transcribing the 
generated data into a form suitable for 
statistical analysis. In contrast, computer 
technology can directly monitor some 
aspects of a subject's performance and 
save the generated data in files that are 
suitable for analysis by a statistical 
package without further processing. 
Unfortunately, it is not currently capable 
of directly providing (for reasonable cost 
and effort) the type of facilities for dealing 
with visual data that are available with 
video. However, the recent merging of 
video and computer technologies with the 
advent of digital video opens up exciting 
possibilities for developing a whole range 
of new research tools that can be used to 
increase our understanding of the way 
people deal with visual information. 
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